


Institutional Archive of the Naval Postgraduate School 


Calhoun: The NPS Institutional Archive 
DSpace Repository 


Theses and Dissertations 1. Thesis and Dissertation Collection, all items 


1977-03 


Decision verification as an element of Navy 
quality assurance programs 


Chadick, Wayne L. 


http://ndl.handle.net/10945/18064 


Downloaded from NPS Archive: Calhoun 


| Calhoun is the Naval Postgraduate School's public access digital repository for 
D U DLEY research materials and institutional publications created by the NPS community. 
get Calhoun is named for Professor of Mathematics Guy K. Calhoun, NPS'‘s first 
KNOX appointed — and published — scholarly author. 





LIBRARY Dudley Knox Library / Naval Postgraduate School 
411 Dyer Road / 1 University Circle 


http://www.nps.edu/library Monterey, California USA 93943 


ono a —— a ——— .2 aya 
wm es — ——a ne Po Ont rap taecer. ae 
epbise neler i Agee a a ay ony rate ’ * 
ANE AeyOh, et Oot pane Meniawrnneee ape ae NM nr) nh pnt i = 
— ortega eae 7 . ye: toe Prd COE P of, hy rete — = 
Sl op Rene Antine _eig etre cha ne eS 


«= 
—— ah Se 
fle age 


: ae : Se 2m wat 
be AX Pains Me dliaiaasthinenh litbeetee een eee = ays oe rs 
2 - : a aioe) ATOR toy In Oy ys ne tet te hag gel sure caetn ae cot 
ata Sr he con arabesmaaeheane 4a tcae ee - 

— te SS emnapaiig Rios = as " 

—— <i > ~ Sawer urenerana tees creme ond 
— a bes oa 5 . x 

= : . a Bann my 

a= P "a 
Co 





—hs 
a | wpe 
_ i 
* 
© 
— 9, Yoottie aa 
4 . 
eas Te senate bees 
“ —_ = | 
Anat 
* 
. = 
. = -~ 
—_~! S 











NAVAL POSTGRADUATE SCHOOL 


Monterey, Galifornia 





THESIS 


DECISION VERIFICATION AS AN ELEMENT 
OF 





NAVY QUALITY ASSURANCE PROGRAMS 


by 
Wayne L. Chadick 


Harel 9/7 


Thesis Advisor: G. F. Lindsay 


Approved for public release; distribution unlimited. 


T1779 44 








SECURITY CLASSIFICATION OF THIS PAGE (When Data Entered) * 
READ INSTRUCTIONS 
— a  —f GOVT ACCESSION NO 3. RECIPIENT'S CATALOG NUMBER 


4 TITLE (and Subtitie) 






5. TYPE OF REPORT & PERIOD COVERED 
Master's Thesis; 
March 1977 





Decision Verification as an Element of 
Navy Quality Assurance Programs 






6. PERFORMING ORG. REPORT NUMBER 





7. AUTHOR(e@) 8. CONTRACT OR GRANT NUMBER(e) 


Wayne L. Chadick 


c F ZAT} AND ADORESS 10. PROGRAM ELEMENT, PROJECT, TASK 
9. PERFORMING ORGAN! ON NAME AN Raed ol ht bet one) 
Naval Postgraduate School 


Monterey, California 93940 
12. REPORT OATE 
March 1977 


13. NUMBER OF PAGES 


5/ 


18. SECURITY CLASS. (o/ thie edport) 





. CONTROLLING OFFICE NAME ANDO ADORESS 


‘ Naval Postgraduate School 
Monterey, California 93940 











. MONITORING AGENCY NAME & ADORESS(If diiferent from Controlling Olfice) 


Naval Postgraduate School 
Monterey, California 93940 






Unclassified 







Sa. DECLASSIFICATION/ DOWNGRADING 1 
SCHEDULE 


. DISTRIBUTION STATEMENT (of thie Report) 


Approved for public release; distribution unlimited. 


. DISTRIBUTION STATEMENT (of the abetract entered in Block 20, if different frem Report) 


: SUPPLEMENTARY NOTES 


= 


- KEY WORDS (Continue on reveree cide if neceeeary and identify by block number) 


Decision Verification 
Quality Assurance 





20. ABSTRACT (Continue on reveree side if necessary and identity by bisek mumbeor) 


Decision verification in quality assurance is simply the Government's 


determination of whether or not a contractor's inspection decisions are 





correct. This paper discusses procedures for accomplishing decision 





verification both as prescribed by directives and as actually done. It 


DD en's; 1473 edition oF | Nov 68 1s oBsoLeTe 


(Page 1) SECURITY CLASSIFICATION OF THIS PAGE (When Data Entered) 
4 





SL CuURMTY CLASSIFICATION OF THIS PAGE(*hen Deta Entered: 


iu 


examines reasons why decision verification does not appear to be a viable 





element in current quality assurance programs even though this procedure 
1s still prescribed in Navy directives for use by contract administrators. 
The use of operating characteristic curves as a managerial aid to deter- 


mine whether or not to use decision verification is also discussed. 


DD Form. 1473 


- an 4 
S/N 0102-014-6601 SECURITY CLASSIFICATION OF THIS PAGE(When Data Entered) 





Approved for public release; distribution unlimited. 


Decision Verification as an Element 
OL 
Navy Quality Assurance Programs 


by 


Wayne L. adick 
Commander, United States Navy 
B. S., United States Naval Academy, 1958 
M.M.A., University of Rhode Island, 1975 


Submitted in pertial fulfillment of the 
requirements for the degree of 


MASTER OF SCIENCE IN MANAGEMENT 


from the 


NAVAL POSTGRADUATE SCHOOL 
March 1977 








ABSTRACT 


Decision verification in quality assurance is simply the Government's 
determination of whether or not a contractor's inspection decisions are 
correct. This paper discusses procedures for accomplishing decision 
verification both as prescribed by directives and as actually done. It 
examines reasons why decision verification does not appear to be a viable 
element in current quality assurance programs even though this procedure 
1S still prescribed in Navy directives for use by contract administrators. 
The use of operating characteristic curves as a managerial aid to deter- 


meme whether Or not to use decision verification is also discussed. 
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I. INTRODUCTION 


Because of the shift in emphasis away from "hands-on" inspection 
by the Government in quality assurance, effective methods of evaluating 
the contractor's quality program have been sought by government agen- 
cles responsible for contract administration. Decision verification is 
one procedure which was developed to assist in assessing the effective- 
ness of a contractor in controlling product quality. 

Decision verification involves the Government's Peter ineion as to 
whether contractor inspection decisions are correct or incorrect. 
Statistical methods are then used to determine if the contractor's in- 
Spection process is effective under criteria established by the Govern- 
METI < 

Decision verification has been applied with varying degrees of in- 
terest, enthusiasm, and success. Some of the Government agencies which 
adopted these procedures in the 1960's have since ceased to use them 
because of real or perceived difficulties. In this paper we shall be 
concerned with what the problems are, and how to overcome them so that 
decision verification can be a eaeual means of judging the inspection 
decision process of the contractor. The basic approach is a 

1. Discuss how decision verification should be accomplished if 
done in accordance with guidance contained in Navy directives. 

2. Discuss how decision verification was observed to be done at 
selected activities, if it was done at all. 

3. Show how the specified decision verification procedures were 
developed, and suggest what could be done to improve the accomplishment 


©r decision verification. 





As an initial step in research, applicable directives, reports, 
and other documents dealing with decision verification were nen 
This provided a basic knowledge of how decision verification should 
be done, at least in theory. Next, visits to several production 
facilities and interviews with quality assurance personnel at these 
facilities and in other Government offices provided information on how 
decision verification really was done and the problems encountered in 
doing it. 

The plan of this paper is to compare the specified with the actual. 
Early discussion centers on the quality assurance policy of the Depart- 
ment of Defense, which caused procedures like decision verification to 
be developed. Then the relationship of decision verification to other 
quality assurance procedures in the overall Navy quality assurance pro- 
gram is discussed. How various decision verification procedures differ 
Within the Navy is also highlighted. This is followed by a discussion 
of the "real" world procedures or how decision verification is actually 
done in the field. The final section of this paper contains conclu- 
Sions and recommendations. 

Now, to set the scene, the next chapter discusses Department of 


Defense quality assurance policy. 





II, DECISION VERIFICATION IN DOD 
QUALITY ASSURANCE PROGRAMS 

Decision verification procedures are used to determine whether or 
not a contractor's inspection decisions are correct or incorrect, and 
with this information in hand, to assess the effectiveness of the con- 
tractor's inspection process. This chapter deals with the application 
of decision verification procedures by the Navy and other organizations 
Within the Department of Defense. The purpose of this chapter is to 
tell why these procedures were developed, and where they fit into the 
quality assurance programs of various organizations within the Depart- 
ment of Defense. This chapter gives some reasons for the emphasis on 
quality assurance by Government procurement organizations and discusses 
DOD quality assurance policy. It outlines the basic elements of quality 
assurance programs of two U. S. Navy organizations and highlights some 


of the differences which exist in these programs. 


A. IMPORTANCE OF QUALITY ASSURANCE 

Quality assurance 1S an important part of systems procurement by 
the Department of Defense. It is easy to cite at least three reasons 
mer this. 

First, many of the current systems purchased by DOD are highly com- 
plex. This is true not only for major systems such as aircraft and 
ships, but also for smaller weapons systems. For example, the MK 56 
sea mine requires 140 individual components for assembly of the completed 
weapon. Effective quality assurance improves the probability that these 


140 components will be assembled into a unit that will work as designed. 








The second reason is closely allied to the first. Frequently, 
many of the individual components needed for the complete system are 
produced by different manufacturers using a variety of processes. 

Many prime and sub-contractors produce items which must function 
together in the end product. For example, over 125 different contrac- 
tors are manufacturing hardware for the NASA Space Shuttle. These con 
ponents and subassemblies range fran simple transformers to such 
complex equipment as the Space Shuttle's main engines [Ref. 1]. Effec- 
tive quality assurance is needed in order to minimize the problems 
asscciated with assembling all of these different components into one 
major vehicle. 

A third reason for the emphasis on quality assurance is the avoid- 
ance of financial waste associated with systems which fail to perform 
as designed. A communications or intelligence satellite program, 
which can cost well over $100 million, may waste substantial resources 
in support equipment and facilities if the space vehicle does not 


survive in orbit as long as intended. 


B. DOD QUALITY ASSURANCE PHILOSOPHY AND POLICY 

The current DOD philosophy on quality assurance was initially de- 
veloped in 1954 and formally expressed in DODINST 4155.6. This instruc- 
tion made contractors responsible for assuring that material submitted 
for acceptance to the Government did meet contract specifications and 
requirements. The contractor had always been responsible for producing 
material to the requirements of the contract, but prior to 1954 it had 
Been up to the Government to determine whether or not the material was 
acceptable. Since acceptability was determined by inspection of the 


material itself, a considerable amount of "hands-on" inspection by the 
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Government was necessary. Beginning in 1954 there was a redirection 
of quality assurance emphasis away from product inspection and toward 
the assessment of the contractor's quality assurance program. The 
Department of Defense no longer performed the extensive product in- 
Sspections which had formerly characterized its approach to quality 
assurance. Instead, the contractor was now responsible for performing 
many of the product inspections previously done by the Government 
quality assurance representative [Ref. 2]. 

The basic DOD procurement quality assurance policy can be found in 
Armed Services Procurement Regulations [Ref. 3]. This document sets 
forth the quality assurance responsibilities of the Government and of 
the contractor, and issues policy guidance for quality assurance rather 
than detailed plans for establishing and operating an effective quality 
program. 

These regulations make it clear that the Government must determine 
the quality requirements which are to be included in the contract. 

This is usually done by contractual provisions dealing with performance 
specifications, product specifications, standard inspection clauses, 

‘ and other quality assurance related specifications such as MI1-Q-9858A, 
Quality Assurance Program Requirements. 

The contractor is assigned responsibility for controlling the . 
quality of his product and for assuring that only contractually conform- 
ing items are delivered to the Government. He must also maintain, and 
furnish to the Government if requested, objective evidence that he is 
fulfilling his quality responsibilities. 

Armed Services Procurement Regulations do not prescribe specific 


qualityv-assurance measures. In accordance with these regulations: 
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It is the contract administration office responsibility to 
develop and apply effective and efficient_procedures for 
Government procurement quality assurance. 
The regulations list a number of possible quality assurance procedures 
from which the responsible contract administration office may choose 
for administering the quality requirements of a given contract. These 
options include, but are not Limited to the following: 

1. Physical inspection of the contract item. 

2. Evaluation and approval or disapproval of the contractor's 
system for controlling quality. 

3. Use of information from consumers to uncover deficiencies. 

Now that Defense Department quality assurance policy has been out- 
lined, we will discuss some of the organizations that carry out that 
policy, with particular emphasis on organizations that administer Navy 
SeMmeraccs. 

Navy contracts are administered by one of several sources. Some 
of these contracts are administered by the office of a Naval Plant 
Representative which is located at the contractor's plant. Others are 
administered by the office of the Supervisor of Shipbuilding, Conver- 
Sion, and Repair located at each Naval District headquarters. Navy con- 
tracts for material which is being produced in plants under the primary 
cognizance of another Service such as the Air Force are normally 
administered by that Service. The Defense Contract Administration 
service often administers contracts for military material production 


in plants where no Service personnel are assigned. 


Ul S. Department of Defense, Armed Services Procurement Regulations, 
(Washington: 1 July 1976), p. 14:8. 
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It is interesting to note that Defense Department agencies includ- 
ing the Navy are sometimes tasked to administer contracts for other 
Government organizations outside of the Department of Defense. For ex- 
ample, the Defense Contract Administration Service has been delegated 
to act for NASA on same Space Shuttle contracts. NASA does not prescribe 
all of the detailed quality assurance actions which the delegated agency 
should take. The Defense Contract Administration Service must use its 
existing procedures as well as any guidance or special instructions which 
NASA provides [Ref. 4]. 

C. SOME DIFFERENCES IN QUALITY ASSURANCE 

PROGRAMS OF VARIOUS AGENCIES 

It has been pointed out that Navy contracts are sometimes adminis- 
tered by other agencies, and that in turn the Navy may be called upon 
to administer other than Navy contracts. Since the contract administra- 
tion activity generally uses its own quality assurance procedures on 
all contracts that it administers, it is important to note some of the 
differences that exist between the quality assurance programs of the 
various DOD agencies. 

Requirements for DOD quality assurance programs are contained in 
Defense Supply Agency Manual 8200.1 of August 1976 which is entitled, 
Procurement Quality Assurance [Ref. 5]. Although this is a Joint Ser 
vice directive and was co-signed by representatives of each military 
Service, the Navy has not yet chosen to abide by this manual without 
issuing its own supplemental instructions. 

In connection with this discussion of decision verification, it is 
interesting to note that one of the predecessors of the present manual 


was issued in 1969. This directive, commonly referred to as Handbook 
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$7, contained a decision verification procedure called "Contractor 
Decision Verification" [Ref. 6]. The requirement to accomplish Contrac- 
tor Decision Verification was cancelled in 1973 when Handbook 57 was 
superseded by an early version of Defense Supply Agency Manual 8200.1. 
Consequently, the Defense Contract Administration Service, which organi- 
zationally is part of the Defense Supply Agency, no longer uses 
Contractor Decision Verification procedures in administering contracts. 
Thus we see that a very important part of the DOD contract administra- 
tion organization omits decision verification as an element in quality 
assurance. 

Unlike the Defense Contract Administration Service, the Navy has 
not eliminated decision verification from its repertory of quality- 
assurance procedures. At least two organizations within the Navy 
Material headquarters have issued quality assurance instructions con- 
taining decision verification procedures to be used by their field 
contract-administration offices. 

The Naval Air Systems Command is one of the two organizations. 

It exercises control over contract administration of production in 
some contractors! plants through the office of a Naval Plant Represen- 
tative or NAVPRO who is physically located at the production facility. 
He 1s responsible for developing and applying effective and efficient 
quality assurance plans and procedures which are consistent with 
requirements of the particular contracts which he acministers. 

The basic elements of the quality assurance program for use by 
contract-administration field activities which are under the cognizance 


of the Naval Air Systems Command are as follows: 
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1. Planning. The NAVPRO initiates plamning as soon as the con- 
tract document is received. Planning includes the selection of the 
applicable elements which are compatible with local production condi- 
tions. 

2. Procedures Review. A review is conducted to confirm that the 
contractor's quality procedures are complete and comprehensive in 
scope. 

3. Procedures Evaluation. A continuing assessment is made of the 
contractor's procedures to confirm that his procedures are adequate 
to assure the quality of the products or services delivered to the 
Government, and that the contractor is actually following these pro- 
cedures. 

4, Product Verification. The actual product item is examined, 
tested, and inspected to determine whether or not the item conforms 
to the contract's requirements. 

5. Contractor Decision Verification. Decision verification pro- 
cedures are a primary topic of this thesis. Contractor Decision 
Verification is defined as "A random and continuing evaluation of the 
contractor's decisions, performed by comparing the findings of Govern- 
Ment inspection with contractor's inspection records .'""* It will be 
applied whenever "in-process product control is performed by.the con- 
tractor and the volume of production and duration of contracts is 


eet 


‘Naval Air Systems Command, NAVAIR FIELD CONTRACT MANUAL, NAVAIRINST 
4330.16 (Washington: Nov. 1971), p. 4-108. 
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6. Quality Data Evaluation. A periodic evaluation of all accum- 
lated quality data 1s made for the specific contract being administered. 
teas data includes such items as contractor inspection records, pro- 
cedures review information, and user feedback on the quality of the 
product. It is used for a number of purposes including adjustment of 
the intensity of Government inspection. 

7. Corrective Action. Corrective action is the action that the 
Government requires the contractor to take in order to correct defi- 
clencies in the product delivered to the Government [Ref. 7]. 

Thus we see that Contractor Decision Verification is one of 
several quality assurance techniques which the NAVPRO can use in his 
quality assurance program. 

Another Navy organization which still retains decision verifi- 
cation procedures as part of its quality assurance program is the Naval 
Sea Systems Command. This command has issued two manuals. One is 
applicable to ship acquisition contracts, and the other manual applies 
to ship repair contracts [Refs. 8,9]. 

The quality program for ship acquisition is similar to that for 
ship repair contracts, but there are significant differences between 
the two. For example, the basic elements which constitute the procure- 
ment quality assurance program for ship repair contracts are also ° 
applicable to the quality assurance program for ship acquisition con- 
tracts, but an additional element is included in the latter. The 
following is a list of the basic elements of both the ship acquisition 
quality program and the ship repair contract quality program. Elements 


common to both programs are indicated by an asterisk. 
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1. Planning.* 

2. Procedures Review.* 

3. Product Verification* 

4, Verification of Contractor's Records. This procedure is in- 
cluded in the quality assurance program for ship acquisition but not 
for ship repair. It is a decision verification procedure similar to 
Contractor Decision Verification procedures previously mentioned in 
the discussion about quality programs of the Naval Air Systems Command. 

5. Procedures Evaluation* 

6. Corrective Action* 

7. Quality Data Evaluation* 

From this listing we see that verification of contractor's records 
is an element of quality assurance programs for ship acquisition, but 
not for ship repair contracts. It should be pointed out here that a 
much stronger emphasis on the independence between Government quality- 
assurance functions and contractor quality functions is found in the 
quality assurance instructions for ship repair contracts, than one can 
find in the quality assurance directives for ship acquisition. The 
following quotation which refers to ship repair contracts illustrates 
his. 

SUPSHIP quality assurance personnel shall not serve as an 
adjunct to, or a replacement for the contractor's own inspection 
system. For example, the SUPSHIP office should not serve as a 
convenient means of advising the contractor of obvious deficiencies 
in work performance; nor should it be used by the contractor as a 


progressive inspection device to determine whether or not the end 
product will be acceptable. 


‘Naval Sea Systems Command, Ship Repair Contracting Manual, 


NAVSEA 0900-LP-079-5010 (Washington: 1974), p. 1202. 
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Besides differences of philosophy within the Naval Air Systems Com- 
mand, there are some specific differences concerning decision verifica- 
tion procedures between the Naval Air Systems Command and the Naval Sea 
Systems Command. For example, verification should be accomplished by 
Naval Air Systems Command contract administration activities, but is 
optional for Naval Sea Systems Command contract administration activities. 

This chapter has shown that decision verification procedures are 
still an element of some agencies' quality assurance programs. This 
along with the fact that various agencies have somewhat different 
quality assurance ‘programs are the two most important points to be 
gleaned from this chapter. The next chapter will discuss the concept 
and objectives of decision verification in a quality assurance program. 
It will also describe specific procedures used by two Navy organizations 


to accomplish decision verification. 
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tite DECISION VERTPICATION, 
CONCEPTS AND PROCEDURES 

This chapter discusses the concept of decision verification and 
the reasons for applying decision verification procedures in the ad- 
ministration of Navy contracts, in order to show why these procedures 
are used and how they are applied. To do this, the initial discus- 
Sion will focus on instances where decision verification can be used 
advantageously. The discussion will then shift to a narrative descrip- 
tion of how decision verification 1s to be accomplished when it is 


done in accordance with Navy directives. 


Pee lie, NEED FOR DECISION VERIFICATION 

In many procurements, there are circumstances in which the examina- 
tion of the end product alone cannot be used to ascertain whether or 
not the item is acceptable and meets contractual requirements. When 
large numbers of identical items such as gun projectiles are manufac- 
tured, 1t may be too costly to inspect each item. When expensive 
items are subject to destructive testing, it may be uneconomical even 
to test large samples of the end item. An aircraft contains large 
numbers of electrical cable runs, some of which are inaccessible for 
normal inspection once the aircraft has been assembled. This wiring 
must be installed a accordance with specifications, but it may not be 
Feasible to perform 100% product inspection on every run of wiring. 

Since extensive product inspection is often impractical, it 
appeared highly desirable to the authors of DOD quality assurance 


programs that the Government be able to determine that the contractor's 
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in-process operations are producing product which is within acceptable 
quality limits. They assumed that if the contractor has an effective 
quality Beetrerce program, his product would meet contractual standards 
most of the time. The contractor's inspection process is one important 
part of his quality assurance program [6]. Therefore it is important 
to know how well this process is working. 

One element of inspection is the actual decision of the inspector 
concerning whether or not a certain attribute or characteristic meets 
contractual requirements. Measuring the effectiveness of a contractor's 
program by an examination of these decisions will be the eee of the 


next section of this chapter. 


B. THE CONCEPT OF DECISION VERIFICATION 

Decision verification procedures originated from a concept developed 
Byeene Air Fores an the late 1950's. By 1963 these procedures had been 
adopted by the Defense Supply Agency and included in its directives on 
quality assurance. 

Under the concept of decision verification, the Government quality 
assurance representative "inspects" the decision made by the contractor's 
inspector. He then determines whether the contractor's inspector made 
a good or a bad decision. The contractor's inspector can make one of 
two basic-errors which result in a wrong decision. The inspector can 
fail to report a real defect,or he can report a false defect. The former 
happens when the inspector fails to see A Guo or if he does see it, 
fails to note it as a defect through an error in judgment. A false 
detection is solely an error in the inspector's judgment. Therefore, 
the QA representative can check on the decision of the inspector using 
both material which 1s accepted by the contractor's inspector and 


material which is rejected [Ref. 10]. 
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The effectiveness of the contractor's inspection decisions can be 
described as a process average representing the percent of decisions 
that are incorrect because of either type of error. Typically, direc- 
tives specify an acceptable quality level (AQL) of 1% for the con- 
tractor's decision process, which means that the Government expects 
that the contractor's inspector will make the correct decision at 
least 99% of the time [6]. 

It is easy to see why the AQL had to be greater than zero, for 
seldom does an inspector always make correct decisions. Apparently the 
1% AQL was chosen after evaluating the quality performance of a large 
number of contractors of various sizes producing various products. 
Useful information as to why an AQL of 1% was selected can be found 
in a 1969 Defense Contract Administration Service lesson plan for Con- 
tractor Decision Verification procedures training [Ref. 11]. Accord- 
ing to this lesson plan, experience indicates that 95 percent of the 
contractors will operate with 0% to 1% process average as related to 
Contractor Decision Verification. 

Decision verification was developed primarily to assure that the 
contractor's quality program or inspection system satisfactorily con- 
trols the quality of the product. This statement has occasionally 
been interpreted incorrectly to mean that if the contractor's decision 
process iS within acceptable limits (1%) no quality defects should be 
found in the end product. In March 1971, the report of a Joint Service 
Panel on Contract Administration strongly questioned the value of 
decision verification. The report said: 

In actual practice, rarely does a contractor's percent defec- 


tiveness exceed the process average limits. As a consequence, in 
some cases when product quality deficiencies have been encountered 
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during product verification, the contractor's decision making 

process is considered acceptable because it has not exceeded the 

process average limits. For this reason the value of CDV as 

currently applied is questioned. 
Overlooked here is the fact that if the contractor's inspection 
system is inadequate, his inspectors could still make predominantly 
good decisions and the end product could still be unsatisfactory. 
Decision verification procedures used by DOD really evaluate the 
effectiveness of the implementation of whatever inspection system 
the contractor has installed. These procedures do not alone evaluate 
the adequacy of that inspection system, but can be used with other 
methods to do so. 
C. NAVAL AIR SYSTEMS COMMAND 

DECISION VERIFICATION PROCEDURES 
Contract administration offices under the Naval Air Systems Com- 

mand have been directed to use a procedure called Contractor Decision 
Verification in programs for evaluating a contractor's quality per- 
formance [7]. Contractor Decision Verification is defined as a 
"system performed by the Government to determine how well or the 
accuracy at which the contractor's overall inspection operation is 
functioning."® This is accomplished by evaluating the accuracy of 
randomly selected contractor's inspection decisions on specific or 
general product quality characteristics. The number and type of 
characteristics to be observed by the Government quality-assurance 


representative in evaluating the contractor's program, depend on the 


"Uy. S. Department of Defense, Report of the Joint AMC/NMC/AFLC/AFSC 
Panel on Contract Administration (Washington: 24 Mar 1971), p. 41. 


eNaval Air Systems Command, Navair Field Contract Administration 


Manual NAVATRINST 4330.16 (Washington: Nov 71), p. 4-105. 
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particular process and product being inspected. Random selection is 
done by use of a random number table. 

Before actually carrying out verification procedures, the Government 
quality assurance representative groups the contractor's inspection 
activities into manageable units, and determines points in the produc- 
tion flow process at which the contractor's inspection decisions should 
be verified. ‘These are called verification stations. Verification 
Stations should be chosen to provide an equal chance for all important 
contractor inspection decisions to be verified while simultaneously re- 
ducing the possibility of undue concentration on some particular process, 
product, or individual inspector. Once the total number of verification 
stations necessary for evaluating the contractor's decision process has 
been determined, they are grouped into Product Control Centers contain- 
ing from three to nine stations each. 

Establishing verification stations and Product Control Centers and 
determining which product characteristics will be used are NAVPRO func- 
tions. The actual procedures to be followed in accomplishing Contractor 
Decision Verification are: 

1. Select one-half, but not less than three, verification stations 
within each PCC using a table of random numbers. Any station or sta- 
tions which have not been verified within three consecutive verification 
cycles are automatically included in the next succeeding cycle. 

2. After selecting verification stations, the QA representative 
verifies that the procedures used by the contractor in product process- 
ing in each Product Control Center conform to contract specifications. 
Any defect noted counts as one defective observation. 

3. The next step is to take 50 observations per day from each Prod- 


uct Control Center until a total of 250 observations have been 
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accumulated there. If there are seven or fewer defective decisions in 
this sample of 250 observations, the contractor's decision process is 
considered to be under control at that center. The critical value of 
seven is based on three sigma limits. If the contractor's process is 
under control, the quality assurance representative shifts to less fre- 
quent sampling as discussed below. 

u. If the contractor's decision process is under control, the 
quality-assurance representative takes 50 observations from each PCC 
Within each subsequent interval of 11 calendar days. (For each 11-day 
interval a new random selection of verification stations is made to 
begin the new cycle.) At this sampling frequency, the process is con- 
Sidered to be under control as long as the percent defective in a 
sample of size 50 is 5.2% or less. If the percent defective exceeds 
5.2% for 50 observations, an additional 50 observations are taken at 
the same verification stations within two days. If the percent defec- 
tive of the total 100 decisions examined exceeds 4.2%, the QA repre- 
sentative reverts to taking 50 observations per day for five days until 
the contractor has taken effective corrective action and the process is 
again under control [7]. It should be noted here, that these pro- 
cedures contain no provisions for using samples of less than 50 obser- 
vations. 

Thus we see that Contractor Decision Verification procedures involve 
an initial step of taking 50 observations per day until 250 have been 
accumulated. Then if the process is considered under control, based 
on criteria set forth in applicable directives, the quality assurance 
representative is allowed to use a smaller sample at greater intervals 


of time as long as the process remains under control. If at any time 
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during this reduced sampling period, the process appears out of control 
the quality assurance representative has prescribed procedures to fol- 

low to insure that adequate sampling is done to confirm whether or not 

the process is in fact out of control. 

D. NAVAL SEA SYSTEMS COMMAND DECISION 

VERIFICATION PROCEDURES 

The decision verification procedures used by Naval Sea Systems Com- 
mand activities are called Verification of Contractor Records [8]. 

They are analogous to Contractor Decision Verification procedures used 
by the Naval Air Systems Command. 

The Naval Sea Systems Command seems to be more flexible in the 
application of decision verification procedures than other organiza- 
tions. It gives the individual contract administration office con- 
Siderable discretion in deciding whether or not to use decision veri- 
fication in administering the quality assurance program associated 
with any particular contract. One important procedural difference 
between the accomplishment of Verification of Contractor Records and 
Contractor Decision Verification should be emphasized. For verifica- 
tion purposes the nominal sample size specified by Naval Air Systems 
Command directives is 50 observations. Naval Sea Systems Command 
directives allow the local quality representative to select the nominal 
sample size. The effect of this coupled with the use of a constant 
percent defective as the basis for rejecting the contractor's process 
will be more fully explored in later sections of this paper. 

The selection of stations for verification and their grouping into 
control centers is essentially the same as done for Contractor Decision 
Verification. The number of verification stations in each control 


center is three to nine. 
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The following describes the procedures for accomplishing Verifica- 
tion of Contractor Records: 

1. Just as is done in Contractor Decision Verification, one-half 
but not less than three, verification stations from each control center 
are selected at random using a table of random numbers. Any station 
or stations which have not been verified within three consecutive veri- 
fication cycles are automatically included in the next succeeding 
cycle, 

2. Also as is done in Contractor Decision Verification, one pro- 
cedural observation will be made for each type of product inspected 
and included in the sample of observations of the applicable control 
center. 

3. From this point in the procedures, the differences between 
Verification of Contractcr Records and Contractor Decision Verification 
become significant. Instead of a specified sample of 50 observations 
per day from each control center, the quality assurance representative 
following decision verification instructions of the Naval Sea Systems 
Command determines locally the size of the daily sample to be taken in 
each control center. When five samples of this established size have 
been accumulated, the contractor's process is accepted as under control, 
at that center if the percent defectives in the total number of obser- 
vations in the five groups is 2.8% or less. 

4, If at this point the contractor's process is in control, the 
quality assurance representative reduces the frequency of sampling. 

He takes one sample of the locally established size within each seven- 
day interval. When the percent defective of this sample exceeds 2.8% 


regardless of the sample size, the quality assurance representative 


Zc 


reverts to sampling as specified in the preceding paragraph, until cor- 
rective action has been taken by the contractor and the decision process 
has been brought under control at the 1% AQL [8]. 

Before proceeding to the next topic it is important to emphasize 
that the allowable percent defective under Verification of Contractor 
Records procedures is a constant 2.8% regardless of the size of the 
sample, while for Contractor Decision Verification the limiting percent 
defective varies with the size of the sample. This will be discussed 
in more detail in Chapter V. 

Pee PrOLENTIAL BENEFITS OF AN EFFECTIVE 

SURVETLLANCE PROCEDURE 

The importance of quality assurance has already been discussed in 
Chapter II. The quality assurance functions of DOD agencies currently 
consume a large portion of contract administration resources, much of 
which is spent on product inspection. Over 9,000 people, 34 percent of 
contract administration personnel in DOD, are assigned quality assurance 
responsibilities. A recent study of DOD contract administration noted 
that the Government was unnecessarily duplicating quality-assurance 
actions of the contractor. Too much effort was being expended on prod- 
uct inspection. For example, in one geographical region studied, 
quality assurance personnel spent 46% of their time doing product inspec- 
tion and only 22% of their time doing surveillance of the contractor's 
quality assurance program [Ref. 12]. 

Decision verification procedures originated from the requirement to 
reduce the need for doing product inspection. They were developed as 
Surveillance methods to aid in determining the effectivenss of the con- 


tractor's quality program. It has been estimated that halving the 
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current production inspection effort, even if the man-years of effort 
in surveillance were doubled, would still result in a 15 to 20 percent 
savings in personnel cost of approximately $15 million annually [12]. 
From the above discussion, it seems that any workable surveillance 
method has merit. That is why Contractor Decision Verification pro- 
cedures and other similar procedures were developed. This chapter has 
shown how these procedures should be accomplished. The following 
Chapter will relate how decision verification has been implemented at 


selected activities. 
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ieee bec lolON VERLEICATION 
AME celebs uae Gia AMInbsS 

The decision verification procedures described by the Naval Air 
Systems Command and Naval Sea Systems Command were discussed in the 
previous chapter, and we now turn our attention to the current appli- 
cation of these procedures where they are employed. (Some contract 
administration offices do not use the decision verification procedures 
which are specified in the directives of the various Naval Systems 
Commands.) The purpose of this chapter is to show the diversity which 
exists in the application of decision verification at several field 
activities. 

The chapter opens with a description of some of the facilities 
Which were visited to gather information on decision verification pro- 
cedures. Then we will describe how these procedures are done at these 
facilities, and relate some views about decision verification as 


expressed by the personnel who must use this quality assurance method. 


wee DEoCRIPTLON OF THE FACILITIES VISITED 

several contract administration offices were visited to obtain 
first-hand information on how decision verification procedures were 
applied as part of on-going quality control programs. We shall refer 
to the activities visited as Facilities A, B, and C. 

Facility A is located at a large manufacturing plant now engaged 
in producing combat jet aircraft for the Navy as well as for foreign 


Countries. This is the only military aircraft being produced at this 
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plant. Although this particular aircraft has been produced by this com- 
pany for many years, production was at a relatively low level at the 
time of the visit. Commercial products and aircraft are also produced 
at the same plant. A resident Naval Plant Representative administers 
Navy contracts at Facility A. 

Facility B is also at a large plant engaged in producing Naval air- 
craft. Both jet and non-jet aircraft are produced for the Navy and 
for foreign countries, and a resident Naval Plant Representative admin- 
isters contracts at this plant. 

Facility C is the office of a Naval District Supervisor of Ship- 
building, Conversion and Repair. The quality assurance personnel 
assigned have responsibilities at several large and small shipyards, in- 
cluding a major yard now constructing two large auxiliary ships for the 
United States Navy. This yard will soon begin building a number of 
small combatants. The various production sites for which this office 
1s responsible include some involved in one-of-a-kind experimental 
model production. 

Other activities visited or contacted by phone included additional 
Naval District SUPSHIP offices, and a plant where NASA contracts are 


administered by the Defense Contract Administration Service. 


B. FINDINGS AT FACILITY A 

The visit to Facility A included interviews with the Naval Plant 
Representative and Government quality assurance personnel. 

The Naval Plant Representative expressed the view that the real 
objective of the Navy quality assurance organization was not to empha- 
size product inspection, but rather to insure the See of the 


contractor's quality program. He was quite aware of Contractor Decision 
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Verification, but he indicated that the procedures as specified by 
Naval Air Systems Command directive, were difficult to follow at this 
plant because of the nature of production. Because of this, he had 
been and was interested in determining two things related to Contractor 
Decision Verification. Since he was often unable to acquire the 
required sample size, he wanted to know how many contractor's inspec- 
tion decisions must be observed in order to obtain useful results. He 
also wanted to know whether or not all of the inspection decisions 
verified should be decisions on military hardware since the directive 
does not specify this. 

Government quality assurance personnel with whom Contractor Deci- 
Sion Verification procedures were discussed generally accepted them 
as reasonable, but thought them excessively complex. They indicated 
that it was difficult to use the procedures under certain production 
conditions such as low volume. Another important point which emerged 
from interviews here, and again in conversations with experienced 
supervisors at other facilities, was that successful use of these pro- 
cedures depended largely upon the motivation and competence of Govern- 
ment quality assurance personnel. Concern was expressed that some 
personnel might not follow decision verification procedures correctly 
in order to avoid personal conflict with contractor personnel. Such 
a situation could arise particularly in Basen of close association 
between the QA representative and the contractor's inspector. It 
appeared that conducting product inspection was much easier from a 
human relations standpoint than passing judgment on the competence of 
individual contractor's inspectors by accepting or rejecting their 


BeCisSions. 
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The Naval Plant Representative at Facility A has had his quality- 
assurance people study the application of Contractor Decision Verifi- 
cation procedures as an element of his quality assurance program. He 
then directed his people in attempting to implement Contractor Decision 
Verification as prescribed in Naval Air Systems Command directives. 
They then studied the results to uncover problems. 

In doing this trial, the quality assurance representative randomly 
selected at least three verification stations from each Product Control 
Center. He then went to those stations to inspect military hardware 
which might be available there. He frequently found that no military 
hardware was ready for inspection, or at least not enough of it to make 
the required 50 observations per day in the control center. 

Auditing verification also presents problems. In actual practice, 
results of any observations that are made are to be recorded on 
DD Form 1711 in accordance with prescribed procedures. The characteris- 
tics inspected are not identified on this form. Instead, the verifica- 
tion station is noted on the form together with the number of observa- 
tions made at each station. This makes it difficult to audit results 
in terms of actual hardware inspected unless a defect is noted, and 
corrective action for that specific hardware deficiency is initiated. 

Facility A currently does not perform decision verification .pro- 
cedures in the manner prescribed by directives because the requirement 
for decision verification can be waived if the volume of production is 
insufficient to support its accomplishment. Locally prepared quality 
assurance directives, Quality Division Operating Procedures, state that 
"statistical" methods of accomplishing decision verification have proven 


unworkable, but recognize the concept of decision verification as a 
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reasonable and important part of procurement quality assurance. Conse- 
quently, modified decision verification procedures have been developed 
for use at Facility A. Under the previously mentioned local directives, 
the quality assurance representative evaluates the correctness of the 
contractor's acceptance decision by verifying one or more products in 
each Product Control Center. The section supervisor determines when the 
total number of verifications is adequate in each case. 

By way of review for a moment, recall that the purpose of decision 
verification is to determine the effectiveness of the contractor's in- 
Spection decisions, and that this effectiveness can be expressed as a 
process average in terms of the percent of decisions that are defective. 
The contractor's decision process average 1s compared with an established 
percent defective allowable limit. If the process average exceeds this 
limit, we say that the contractor's inspection decision process is out 
of control, and the contractor will be directed to take corrective action 
to bring his process under control. 

Returning now to the discussion of Facility A, it is emphasized that 
decision verification procedures have not been used at Facility A to 
develop a process average for use in controlling the contractor's decision 
inspection process; rather the results of inspections are related to 
product quality. Requests to the contractor for corrective action have 
apparently been based on the discovery of individual material defects 
during inspection rather than on a determination that the contractor's 
inspection decision process is out of control. If this is the case, it 
Supports the findings of the Joint Service Panel on Contract Administra- 


tion, whose report of March 1971 said: 
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Further, it appears to the Panel, that as constituted, the CDV 
procedures serve only to assure the correction of specific defi- 
ciencies detected solely on a random basis. Thus, if a QAR in 
performing 50 random observations detects a particular product 
or process deficiency, he may direct corrective action to the 
particular deficiency which he encountered. The QAR therefore has 
little way of knowing that other characteristics have turned de- 
defective until this point turns up in a later inspection. 

All of this seems to add up to the fact that Contractor Decision Verifi- 
cation Procedures are being used as a product inspection technique 


rather than as a method of controlling the contractor's decision process. 


C. FINDINGS AT FACILITY B 

The visit to Facility B included interviews with Government QA per- 
sonnel but not the Naval Plant Representative. 

Personnel with whom decision verification was discussed appeared to 
be enthusiastic about this concept as a quality assurance tool, but it 
was soon determined that Contractor Decision Verification was not done 
continually at Facility B. It is done only during periodic quality- 
assurance audits. This appears primarily to result from a lack of 
sufficient personnel to accomplish See verification continuously 
at the nine control centers which have been established. 

Quality assurance audits are undertaken on an as needed basis such 
as when quality data indicates an adverse quality trend developing or 
on a 180 day cycle. During the audit, one quality assurance representa- 
tive works full time on decision verification. 

' Like Facility A, the contractor at Facility B also has cammercial 
production in progress along with military production in the same 
general location. When Decision Verification is done during quality 


assurance audits, only military hardward is used if enough of it is 


TReport of Joint AMC/NMC/AFLC/AFSC Panel on Contract Administration, 
p. 41. 
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available. Quality assurance personnel attempt to stay away from any 
location which is primarily commercial production. If an item of com 
mercial hardware does happen to came through a predominantly military 
product station, and no military material is available for inspection, 
the Quality Assurance Representative is permitted to use the commer- 


cial product for decision verification purposes. 


D. FINDINGS AT FACILITY C 

Facility C is a Naval District Supervisor of Shipbuilding, Conver- 
Sion and Repair office, operating under the direction of the Naval Sea 
Systems Command. The field visit to Facility C was somewhat limited 
in scope, but interviews with QA supervisory personnel were conducted. 

Quality assurance management personnel at Facility C are familiar 
with the decision verification procedure prescribed by the Naval Sea 
systems Command. Recall that this procedure is called Verification of 
Contractor Records, and that the Naval Sea Systems Command makes the 
inclusion of this procedure in any quality assurance program optional. 
At Facility C the quality assurance personnel have elected not to use 
decision verification, although their internal directives contain com- 
prehensive instructions dealing with the accomplishment of Verification 
Seecontractor Records. 

Construction of two large auxiliary ships for the Navy ina 
Civilian yard is one of the major production activities now under the 
cognizance of this office. This construction provides an opportunity 
for application of decision verification procedures. At the highest 
level of construction activity, 16 quality assurance people will be 
assigned directly to the project office with additional people available 


as needed from the Naval District SUPSHIP office. The management plan 
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for these ships specifies quality assurance tasks to be performed. 
These tasks include review and evaluation of contractor's procedures 
and processes, and product verification of hardware, but decision 
verification is not included. 

Facility C personnel gave two reasons for not using Verification 
of Contractor Records procedures. First, the type of operations in- 
volved in ship construction make it difficult to establish control 
centers and verification stations. There often is not enough hardware 
at a given Seen to permit the required number of observations. 
Second, the geographical distribution of activities in a shipyard 
makes it difficult for Government representatives to be present at 
any given time when contractor-inspected material is available to 
be used for decision verification. The shortage of personnel exists 
at other facilities as well. There are just not enough personnel to 


adequately carry out decision verification procedures. 


E. NASA AND DECISION VERLFICATION 

The final field visit was to a contractor's plant in which a NASA 
Space Shuttle is being built. At this plant, the Defense Contract 
Administration Service exercises the contract administration function 
under a letter of delegation from NASA. According to NASA directives, 
the delegated agency 1s to use its own administrative, operational, 
and procedural instructions, and technical documents plus any addi- 
tional direction which NASA provides. If the agency instructions con- 
flict with NASA directives, the latter prevail [4]. 

For inspection purposes there are two important categories of 


characteristics mentioned in NASA contracts. These are "mandatory" 
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and "other than mandatory". A mandatory characteristic is one, which 
if defective, could prevent the article from performing its intended 
purpose in the next higher assembly or result in hazardous or unsafe 
conditions. "Other than mandatory" characteristics are those selected 
for inspection and test "as necessary to prevent degradation of 
quality" [4]. Each mandatory characteristic 1s subject to inspection 
or test on every article of product. "Other than mandatory" 
characteristics may be inspected by sampling when appropriate. 

At the plant visited, the contract is administered by the Defense 
Contract Administration Service which has long since discontinued use 
of decision verification. However, NASA contracts could also be 
administered by Navy offices which still include decision verification 
procedures as an element in quality assurance. NASA's policy on this 
1s: 

The use of "Contractor Decision Verification (CDv)" or similar 
systems as a substitute for "mandatory" or "other than mandatory" 
inspection of an article is prohibited. 

Contractor Decision Verification (CDV) or any similar system 
is not to be substituted for this requirement nor is it to be 
used for product inspection or article acceptance. 

NASA policy as stated in the foregoing quotations has been inter- 
preted by some people to mean that CDV or similar procedures will not 
be used in administration of NASA contracts. 


Obviously the four activities contacted are only a limited sample 


of the total. Nevertheless they are representative of the variety 


eNational Aeronautics and Space Administration, Quality Assurance 
Provisions for Government Agencies, NHB5300.4(2B) (Washington: Nov. 
Moy), p. o-3. 
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of production activity under various contract administration offices. 
This chapter has shown that at least scme activities decision veri- 
fication is not used. The following chapter will discuss these veri- 


fication procedures from the manager's aspect. 
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Ve. SOME COMMENTS FOR CONTRACT ADMINISTRATORS 
ON USING DECISION VERIFICATION PROCEDURES 

Previous chapters have discussed specified procedures for determin- 
ing whether or not a contractor's inspection decision process is oper- 
ating within Government established limits. This chapter discusses 
some ways a Government contract administrator might locally modify the 
specified procedures in order to make then more useful in situations 
where there is too little production volume to feasibly obtain the 
Specified number of sample observations. ere characteristic 
curves are presented to show how variables such as sample size, fre- 
quency of sampling, and actual process fraction defective affect the 
usefulness of decision verification. Also discussed are some human 
factors which impact on the operation of decision verification. 
A. SAMPLING AND DEVELOPMENT OF CONTROL 

CHARTS FOR VERIFICATION 

Since 100 percent inspection of each item is often not practical or 
not considered cost effective, Navy quality assurance personnel use 
sampling techniques in performing decision verification. When sampling 
1s done, there is always a mee that ee ee inspection decision 
performance will be accepted by the Government. This er arises at 
the fact that the sample may not be representative of the process quality 
and from errors in inspection of decisions from the sample. 

In practice, not even 100% inspection always insures a defect free 
product or process. Experiments with 100% inspection indicate that the 


average number of defects detected may be considerably less than 100% 
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of the defects present. An experiment which was done to determine the 
influence of complexity on inspection errors resulted in defect detec- 
tions of from 30% for complex items to 80% for less complex items [10]. 
This of course refers to individual inspectors rather than to the con- 
tractor's inspection process as a whole. The contractor's process 
would be expected to have a higher effectiveness because of repetitive 
inspections and other controls on quality. 

Since 100% inspection may not guarantee that all defective decisions 
Will be discovered, the manager should realize that sampling is even 
less Likely to uncover all errors. It follows that since decision veri- 
fication uses Sampling, we should expect that not all incorrect decisions 
of a cmtractor will be discovered. 

Although the two verification methods discussed in this paper use 
sampling, there are important differences between them. One method is 
based on using a control chart approach to determine when the contractor's 
decision process is out of control.” The other method appears simply to 
sample the inspection decisions, and then accept or reject the contrac- 
tor's inspection process by comparing the percent defective in the 
sample with 2.8%. 

Contractor Decision Verification procedures are based on the concept 
of a control chart called a "p-chart" which is designed to determine 


whether or not the process is under control. The p-chart shows variation 


othe control chart is a useful device for determining whether ob- 
served changes are due simply to chance fluctuations or to actual 
changes in the process because of sucn things as deterioration of 
machine parts, or in this case to mistakes by the inspector. A com- 
prehensive discussion of control charts can be found in Quality Control 
and Industrial Statistics by A. J. Duncan [Ref. 13]. 
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around a central value of the fraction defective in the output of a 
process. In this case it shows variations in the fraction defective 
of inspection decisions. 

Under the assumption that most contractor's inspection decision 
processes are accurate 99% of the time, the Government has established 
an Acceptable Quality Level (AQL) of 1% for contractor inspection 
decisions [11]. This AQL value is used as the central line of the © 
p-charts for each Product Control Center when Contractor Decision Veri- 
fication 1S applied as an element of a quality assurance program [7]. 
The Upper Control Limit (UCL) for the chart is set at three standard 
deviations, above the AQL, or the so-called "three-sigma limit". The 
use of a three-sigma limit significantly reduces the probability that 
any given sample result which falls outside this limit does so by 
chance alone if the process is under control. 

In order to illustrate how the p-chart could be prepared using Navy 
procedures for Contractor Decision Verification, we consider the 
initial step in verification of taking 50 observations for five days 
to get a total of 250 observations. The central line fraction defective 
p' is the AQL value of 0.01. In the example, the three-sigma limit can 


be camputed as follows: 


Pee pra") : ss 2 ORLY ana 


Three-sigma = 0.018 


The Upper Control Limit for this p-chart is equal to the central 
line value of 0.01 plus the value of three-sigma, and thus the Upper 


Control Limit in this case is 0.028. The acceptance number for the 
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sample of 250 is obtained by multiplying the sample size by the value 
of the Upper Control Limt. The acceptance number for the example is 
7. Thus if 7 or fewer defective decisions are found in a sample of 
250, the contractor's inspection decision making process is considered 
to be under control. 

Continuing with this example, if the process is under control after 
the initial sample, only 50 observations are required each 11 day 
interval thereafter as long as the process remains under control [7]. 
The p-chart for the 50 observations 1s similar to the previous chart. 
The central line is at 0.01, but the control limit is different. Since 
the standard deviation varies inversely with the square root of the 
sample size, the Upper Control Limit gets further away from the central 
line value as the sample size is decreased. In this case the Upper 
Control Limit is now 0.052. Based on this upper limit, the process is 
considered under control unless there are more than three defectives 
in a sample of 50 observations. 

Contractor Decision Verification procedures and Verification of 
Contractor Records have the same objective, but there are significant 
differences in these procedures. We have seen that Contractor Decision 
Verification procedures make use of the concepts of a p-chart and that 
the maximum allowable fraction defective of the sample varies with the 
sample size. This is not true with procedures for Verification of 
Contractor's Records, which do not make use of control charts nor vary 
the maximum allowable fraction defective with sample size. 

In accomplishing Verification of Contractor Records, a sample of 
unspecified size is taken from the contractor's inspection decisions. 


Regardless of the sample size, the contractor's process is considered 
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unacceptable if the fraction defective of the sample exceeds 2.8%. 
Using these procedures, the individual contract administrator has 
considerable flexibility in selecting a sample size commensurate with 
the volume of inspection decisions available instead of being re- 
stricted to samples of 50, 100, or 250 as in Contractor Decision Veri- 
fication procedures. 

This section has briefly discussed how sample results may be used 
in making the decision to accept or reject the contractor's process, 
but it has not given the manager any guidance for application of 
decision verification procedures. The next section will discuss one 
tool the contract administration manager can use in deciding whether 
or not a modified decision verification procedure could be productively 
included in his quality assurance program, even if any requirement to 
accomplish decision verification had been waived because of lack of 
adequate production volume. 

B. OPERATING CHARACTERISTIC CURVES 

FOR DECISION VERIFICATION 

In determining the applicability and feasibility of decision veri- 
fication in his particular situation, a manager should consider a 
number of factors. These include practical sample size, frequency of 
sampling, cost of inspection versus cost of material rejected further 
along in the process, and other inspections, such as product verifica- 
tion inspection, which may reduce the need for decision verification. 

An operating characteristic curve is one device available to the 
manager as an aid in determining the potential value of decision 


verification procedures in his overall quality assurance program. The 


43 





operating characteristic curve shows how the probability of acceptance 
varies with the quality of decision.*9 7 

Figure 1 shows operating characteristic curves for p-charts which 
could be used in Contractor Decision Verification Procedures. Three 
of these curves, namely 50, 100, and 250 are those which should be used 
when doing verification as specified by the directive. Each curve is 
plotted for a different sample size. The probability of the results of 
the verification procedure being considered acceptable for a sample of 
Size (n) is plotted against the process fraction defective. 

Turning to Figure 1 it can be seen that for any particular value of 
process fraction defective, with an AQL of 0.01, a sample size of 25 
gives less probability of detecting an out of control process than a 
sample size of 50. The manager can use the operating characteristic 
curves to determine how much better chance of detecting a shift in the 
process exists when the sample is 50 instead of 25. Doubling the sample 
size generally does not double the probability of detecting a process 
shift. The manager should compare the value of an added increment of 
protection against undetected process shifts to the cost of getting that 
additional protection. 

Continuing with Figure 1, if the actual fraction detect eset tne 
process shifts to 5%, there is about a 0.14 probability that this shift 


in the process will be detected if a sample of 25 is taken. Doubling 


e For a detailed discussion of operating characteristic curves, see 
Quality Control and Industrial Statistics by A. J. Duncan. Chapter 7 
and Chapter 19 of this text are of particular value in understanding 
operating characteristic curves applicable to decision verification 
procedures. 
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Figure 1. Operating characteristic curves for Single limit p-charts 
for Contractor Decision Verification. 
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the sample size to 50, gives a 0.24 probability that the process shift 
will be detected. If these same sample sizes are used to verify the 
process at 11 day intervals as specified in procedures for Contractor 
Decision Verification, it is possible to calculate that there is an 
approximate 0.60 probability that the shift will still be undetected 
after about 30 days if the sample size is 25. This probability drops 
to approximately 0.40 if the sample size is 50. Since the samples are 
random, the probability of not detecting a shift after a given number 
of samples is the product of the individual sample probabilities of 
not detecting the shift. 

Figure 2 shows operating characteristics curves for sample inspec- 
tion plans which demonstrate the performance of the Verification of 
Contractor Records procedures where the sample size is arbitrary. 
Curves for amit sample sizes are shown. It can readily be seen 
again that the sample size affects the probability of accepting the 
contractor's inspection process for any particular process fraction 
Bepecelve.. 

The manager can use these curves in much the same way as he could 
use those in Figure 1. From the various curves he can determine how 
the probability of accepting the contractor's aaeee varies with 
sample size. 

It is interesting to compare some operating characteristic curves 
for Contractor Decision Verification and those developed here for use 
in Verification of Contractor Records. Figure 3 shows some operating 
characteristic curves for Contractor Decision Verification and for 
Verification of Contractor Records for the same sample sizes. It can 


easily be seen that when both approaches use the same sampling size, 
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Figure 2. Operating characteristic curves for single sample inspection 
plans for Verification of Contractor Records. 
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Comparison of some operating characteristic curves for 
Contractor Decision Verification and for Verification 
of Contractor Records. 
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Verification of Contractor Records procedures give a greater probability 
of rejecting the contractor's process for unacceptable process fraction 
defectives. This is true up to a sample size of 250 at which point the 
operating characteristic curves for Contractor Decision Verification 

and for Verification of Contractor Records are identical. This means 
that for samples of less than 250 observations, the decision verification 
procedures used by activities under the Naval Sea Systems Conmand perform 
the decision verification function with greater statistical effectiveness 
than procedures specified by Naval Air Systems Command. 

In spite of the fact that elther of the decision verification pro- 
cedures discussed in this paper can be useful in a quality assurance 
program neither of them was found to be extensively used. The next 
section discusses some possible reasons for this non-use. 

C. PRACTICAL PROBLEMS OBSTRUCTING THE 

USE OF DECISION VERIFICATION 

Decision verification procedures were developed to aid the Government 
in measuring the effectiveness of a contractor's quality assurance pro- 
gram. In practice these procedures have evidently not produced the 
desired results. They have been criticized and even deleted from some 
quality assurance manuals. 

People are the focus of many problems associated with decision veri~ 
fication. In many cases there are not enough quality assurance personnel 
to do all of the assigned tasks, including decision verification. If 
enough confidence can be generated in decision verification, a manage- 
ment decision might result in more time spent on this type of inspection. 
Quite the contrary appears to be happening today. 

The value and the feasibility of decision verification procedures 


is being seriously questioned by the users of these procedures and by 
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various study panels on procurement. There is some merit to the argu- 
ments against decision verification. For example, looking at the 
operating characteristic curve in Figure 1 for a sample size of 250, 
one sees that a process fraction defective three times as large as the 
Acceptable Quality Level has approximately a 50% chance of being 
accepted. It is little wonder that some people have lost confidence 
in decision verification. On the other hand decision verification in 
some cases may offer all the assurance of inspection quality necessary. 
For example, using the same sample size of 250, if a manager is willing 
to accept some probability that the fraction defective can be as much 
as but not exceed a fraction defective of 0.06, he may confidently use 
decision verification procedures, since the operating characteristic 
curve shows that for a sample of 250, the probability is almost zero 
of accepting decision processes which have fractions defective of 0.06 
Se ereater. 

There is some concern about the motivation of quality assurance 
personnel to do accurate decision verification. Several experienced 
Government supervisory personnel with whom this was discussed felt 
that the quality assurance representative was often reluctant to per- 
form decision verification because he had to judge the performance of 
personnel rather than performance of a manufacturing process. This 
problem could be especially acute in plants where Government quality 
assurance personnel have had long association with the contractor's 
inspectors. 

If a quality assurance representative is less than conscientious 
he can easily falsify his records of decision verification performance. 
The DD 1711 form used to record verification lists the stations veri- 


fied, but specific hardware items inspected are not identified unless 
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a defective observation is made. The audit trail for decision verifi- 
cation 1s inadequate. Thus we see that attempts to accomplish decision 
verification may well have to overcome several obstacles, not the least 
of which is in some cases the possible lack of motivated and energetic 
quality assurance representatives. 

This chapter has discussed the idea of control charts used in veri- 
fication, and has noted differences in two Navy decision verification 
procedures. The value of operating characteristic curves to the 
manager has been illustrated. The final chapter of this paper will 


present conclusions and recommendations arising from this research. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 


Several conclusions have been reached as a result of this investi- 
gation into decision verification procedures used by various contract 
administration offices. These can be conveniently grouped for dis- 


cussion into general and specific categories. 


A. GENERAL CONCLUSIONS 

Some method of assessing the effectiveness of a contractor's inspec- 
tion decisions would be useful in the overall Government quality 
assurance program, but it is not an essential element. The objectives 
of the Government's quality program can be, and most often are, met 
without recourse to decision verification. There is little emphasis on 
decision verification in the performance of many Navy contracts. 

Although statistically sound, decision verification has been sub- 
jected to some analysis in the past [Ref. 14]. It is, however, the 
practical problems associated with accomplishing decision verification 
rather than anything else which have caused decision verification to 


"be neglected. 


Pee or eClLELIC CONCLUSIONS 

Quality assurance directives issued by Naval Air Systems Command 
and Naval Sea Systems Command for use by field contract administration 
activities do not explain decision verification in sufficient statisti- 
cal and probabilistic detail. They do not give the manager enough 
information about decision verification procedures to enable him to 


make a sound decision on whether to use these procedures as prescribed; 
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to modify them to accommodate local conditions; or not to use decision 
verification at all. 

Some activities do not use decision verification procedures con- 
tinuously because there is not enough production volume to support con- 
tinuous verification. Instead, they only include decision verification 
in the quality assurance audit program. Product Control Centers may 
be audited for 10 day periods at approximately six-month intervals. 

Audits can produce little useful information as far as decision 
verification is concerned. As an example of this, if 40 observations 
a day are made for two weeks, during the audit there is about a 30% 
probability that the contractor's process would be accepted as in 
control even if the process fraction defective were 3% instead of the 
Acceptable Quality Level value of 1%. If low production volume was 
the initial cause for doing decision verification only on an audit 
basis, it is doubtful that even as many as 40 observations per day 
would be available. Smaller samples than 40 observations would give 
an even greater probability that the process was in control when it 
actually was not. 

People are definitely a critical element in the practical appli- 
cation of decision verification. Both quality and quantity impact 
on whether or not decision verification will be successful. The 
quality assurance representatives must actually perform the verifica- 
tion, and the eee and managers must have confidence in the 
results. This investigation suggests that in some cases neither event 
occurs, because results of decision verification as currently recorded 


do not lend themselves to easy audit. 
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C. RECOMMENDATIONS 

Because of problems encountered in performing decision verification 
and lack of confidence in the results, there seems to be mixed interest 
among quality assurance personnel in continuing to use decision verifi- 
cation in a quality assurance program. Perhaps the usefulness and re- 
quirement for decision verification should be reviewed by the material 
commands of the Navy and a determination made as to whether or not this 
decision verification should be retained as an element of Navy quality 
assurance. 

Further research is needed to determine the optimum mix of the 
various elements of quality assurance such as product verification, 
procedures evaluation, and decision verification. The possibility of 
developing a more flexible procedure for use by the manager in applying 
decision verification should be investigated. This method might include 
Such variables as volume of production, number of personnel available, 
value of the product, and best location of various Product Control 
Centers in the production flow. 

There are some recommendations of a more immediate nature. For in- 
stance, activities which now accomplish decision verification only on 
an audit basis might consider discontinuing this practice. Because 
of the infrequent sampling, the information obtained about the status 
of the contractor's inspection decision process does not appear to be 
worth the full time effort of a quality assurance representative during 
the period of the audit. 

It is also suggested that decision verification procedures be 
standardized throughout the Navy if they are retained as part of the 


Navy quality assurance program. 
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Finally it is proposed that managers at the local plant level 
have their people develop operating characteristic curves such as 
those in this paper. These curves can then be used in determining 
the value of doing verification using various sample sizes and 
sampling frequencies which are feasible at the individual production 
activity. The local manager should decide whether or not he can 
effectively use decision verification procedures. 

The principal usefulness of this work is in bringing into focus 
the issues to consider and assess concerning the viability of decision 
verification processes at a given plant. 

In this thesis we have discussed some methods of accomplishing 
decision verification as related to quality assurance. We have also 
looked at some of the problemswhich contract administrators face when 
trying to accomplish decision verification under conditions which are 
not favorable to using this quality assurance technique. 

It is hoped that this thesis effort will be useful to those con- 
tract administrators who wish to include decision verification as an 


element in their quality assurance program. 
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